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INTRODUCTION 



In June, 1986, the Office of Special Education and 
Rehabilitative Services, US Department of Education awarded 
the Vocational Education Program at California State Univer- 
sity, Long Beach a three-year special projects training 
grant. The purpose of this project is to design and field- 
test a training curriculum in the area of curriculum-based 
vocational assessment for students with handicaps in 
secondary education.. The overall format of this grant 
includes the development and validation of the curriculum 
sequence and associated personnel competencies in the first 
year; field-testing of the curriculum sequence in both 
in-service and preservice settings during the first and 
second years; and revisions and dissemination of the 
curriculum sequence in the third year. 

Curriculum-based assessment is a relatively recent 
approach to the vocational assessment of students with 
handicaps. Interest in this approach heightened substan- 
tially with the passage of the Carl D. Perkins Vocational 
Education Act of 1984, which stipulated among its mandated 
services for handicapped students enrolled in vocational 
education programs , "an assessment of . . . interests, 
abilities and special needs with respect to successfully 



completing the vocational education program" [Section 
204(c)(2)]. While the merits of a curriculum-based approach 
to vocational, assessment have been elaborated elsewhere 
(Cobb, 1983; Cobb & Larkin, 1985; lanacone S Leconte, 1986; 
Peterson, 1985; Stodden & Boone, 1986; Stodden & lanacone, 
1986), limited attention has been devoted to the training 
of vocational education and special services personnel 
in its use* Hence, the rationale for this product 
development and training project. 

Project Overview 

A series of eight training modules was developed during 
the first project year (June, 1986-May, 1987). The initial 
two modules introduce a rationale for curriculum-based 
vocational assessment, present information about technical 
concepts in assessment, and describe the skills needed 
to establish a vocational assessment process. The remaining 
modules provide instruction on the use of curriculum-based 
assessment procedures in the placement and planning phases 
of a student's program, during the srudent's participation 
in vocational education, and as the student exits from 
the program and moves into the workplace • Procedures for 
evaluating the curriculum-based vocational assessment 
process are contained in the last module of the series . 

Field-testing of the training modules commenced in 
a variety of in-service and preservice settings during 



the last quarter of the first project year and will be 
completed at the end of the second year (May, 1988). The 
in-service f ield--test program was conducted from April, 
1987 through January, 1988 with a total of 38 vocational 
education and special services personnel at three sites: 
(1) Chisago Lakes School System, Minnesota (N = 12); (2) 
Monroe Comprehensive High School, Georgia (N = 12); and 
(3) North Orange County Regional Occupational Program, 
California (N = 13). Each of these field-test programs 
was organized and delivered by faculty from nearby 
universities: Bemidji State University, the University 
of Georgia and California State University, Long Beach, 
respectively. Field-testing of the training curriculum 
in preservice courses at six universities began in June , 
1987 and will conclude in May, 1988 (est. participant N 
= 90). The six universities include the three just 
mentioned, plus George Washington University, University 
of Northern Colorado and the University of Vermont. During 
both in-service and preservice field-testing, evaluative 
information on training products and processes is being 
collected from a variety of sources, including field site 
trainees and trainers, external reviewers and project staff. 

In the third project year (June, 1988 - May, 1989 ), 
three interrelated activities are scheduled to occur. 
First, the entire training curriculum will be revised f 



based on evaluative data obtained from preservice and in- 
service testing. Second, the revised products will be 
disseminated at six workshops: two workshops will be 
associated with presessions held at the annual conventions 
of the American Vocational Association and the Council 
for Exceptional Children and four regional workshops will 
be conducted by faculty at Bemidji State University, George 
Washington University, University of Georgia, and University 
of Northern Colorado. Third, the series of eight modules 
and an accompanying trainer's manual will be marketed and 
disseminated by at least one, and possibly two major 
professional organizations in vocational education and 
special education. For additional description of the 
conceptual and operational dimensions of this 
curriculum-based vocational assessment training project, 
see Albright & Cobb (1987). 

Purpose 

The purpose of this paper is to describe the formative 
evaluation process being used to develop and refine the 
training curriculum. This process is based on a three-phase 
view of program evaluation: 

1. Planning evaluation , which is concerned with program 
design considerations; 

2. Process evaluation , which focuses on implementation 
aspects of a program; and 



3. Impact evaluation / which examines the influence 
of the program on its graduates and their respective work 
environments . 

Similar conceptual presentations of program evaluation 
have been presented by a variety of authors, including 
Sanders and Cunningham (1974), Phelps (1976), Skrtic, 
Knowlton, and Clark (1979), Albright and Markel (1982), 
Brinkerhoff, Brethower, fUuchyj, and Nowakowski (1983), 
Maher and Bennett (1984), and Osigweh (1986). However, 
descriptions of use of this three-phase conceptual frame- 
work in an applied training context are not as apparent 
in the literature. 

A description of each of the three formative evaluation 
phases follows. It must be noted, however, that at the 
time of preparing this paper, the three-year project was 
roughly 60% completed. Phase I planning evaluation 
activities had occurred; Phase II process evaluation 
activities were in progress; and Phase III impact evaluation 
was still in the design stage. Consequently, the following 
reporting will essentially " reflect a mid-point analysis. 
A detailed accounting of Phase I activities and outcomes 
will be provided. The Phase II description will focus 
primarily on data collection procedures and instrumentation. 
Since Phase III is scheduled to occur in the upcoming 1988-8 9 
academic year, this presentation will necessarily be limited 
to the design elements of impact evaluation activities 
that have been completed thus far. 



Phase I: Planning Evaluation 

Introduction 

Planning evaluation involves an assessment of the 
overall conceptual integrity of a system, model or product. 
As noted by Brinkerhoff (1980), the primary purposes of 
planning evaluation in a personnel preparation context 
are: (a) to determine the proper goals of the program, 
and (b) to help in selecting the most appropriate strategy 
for achieving these goals. 

Planning evaluation activities are most visible in 
the early stages of a program or project, when staff efforts 
are directed to program design matters. However, these 
activities should also be present, though perhaps less 
apparent, in the ongoing operation of a program. 

Method 

Three major planning evaluation activities occurred 
at the beginning of this project. Each is summarized below: 
1. Development of Conceptual Framework 

The content of the training curriculum is based 
on a three-stage view of the assessment process; that 
is, assessments are to occur prior to student entry 
into a vocational program, during the student •s parti- 
cipation in the program, and as the student exits 



the program and enters the workplace. This 
conceptualizati n, which was adapted from earlier 
works (e.g., Albright, Fabac & Evans, 1978; Phelps 
& Wentling, 1977; ffitlington, 1981), was initially 
presented in the grant proposal submitted to the US 
Department of Education and subsequently refined during 
the start-up of the project. The refined model 
identifies the various purposes of assessment, the 
key questions to be addressed, and the timing of 
assessment activity in each of the three stages. 
A description of this model appears in Albright (1987) 
and Albright & Cobb (1987, 1988). 

The next step in model development was to place 
it within a personnel training mode. This involved 
the identification of competencies needed by different 
personnel in designing and implementing a curriculum- 
based assessment process. The procedures used to 
co:nplete this step are described in the following 
section. 

2- Identification and Verification of Model Components 
for Training Curriculum 
An initial series of nine training module titles 
was delineated from the three-stage assessment model. 
These titles essentially followed the major purposes 
of the assessment process (e.g., planning the student^s 
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program, monitoring student progress). Within each 
of these module titles or clusters, a set of personnel 
competency statements was derived by the project staff. 

In October, 1986, 16 members of an expert review 
panel were asked to critique the proposed training 
program, which included a description of the three- 
stage assessment model, a list of training module 
titl3s and corresponding personnel competencies . 
The panel members represented nationally recognized 
leaders in vocational special education who have 
conducted personnel preparation programs and research 
in the area of vocational assessment for individuals 
with handicapping conditions. 

The expert review activity was conducted it two 
levels. On a nicro-review level, ten panelists were 
asked to rate personnel competencies and roles within 
selected module clusters. At a mctcro-review level, 
six members were instructed to critique all training 
clusters and associated personnel competencies and 
roles. This micro- and macro-approach was used to 
obtain indepth feedback on individual modules and 
on the overall training system. 

Separate instruments were developed for the micro- 
and macro-reviews. However, both instruments consisted 
of forced-choice and open-ended items on the importance 
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of the various competencies and the roles of vocational 
education and special services personnel in implementing 
selected assessment functions. Figure 1 provides 
an illustration of a survey instrument used in a micro- 
review, 

****** 

Insert Figure 1 about here 
*************************** 

Thirteen (13 ) of the 16 panelists responded to 
the survey. Five members provided macro-review informa- 
tion and eight members -produced micro-review data. 

The information shown in Table 1 is a display 
of the planning evaluation framework used in designing 
the expert panel review activity. 

************************* 

Insert Table 1 about here 
************************* 

3. Review of Training Program Content and Processes 

In January, 1987, a two-day planning meeting of 
the project staff and the trainers of the forthcoming 
in-service field-test program was held at the University 
of Georgia. The purposes of this meeting were to: 

• review the results from the expert panel critique 
of the proposed module system and training compe- 
tencies ; 

• determine the suggested content and format for 
the narrative sections for each of the training 
modules; 
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• establish timelines for completion of draft modules 
and personnel writing responsibilities; 

• review evaluation instruments anc. procedures; and 

• discuss field-test arrangements for the three sites. 
A summary of key decisions made at this meeting was 
prepare^.^ and distributed to personnel with specific 
module writing responsibilities and to the in-service 
trainers. 

Data AnalysiiS and Interpretation 

A summary analysis of the micro- and macro-reviews 
was compiled by the project staff. Figure 2 illustrates 
the format used in analyzing these data and for communicating 
decisions made relative to module revisions. 

*************************** 

Insert Figure 2 about here 
*************************** 

The information presented in this summary analysis pertained 
to the content and sequence of the training curriculum. 
In addition, a separate summary analysis was completed 
regarding --.he expert review panel's recommendations for 
the involvement or roles of various personnel in specific 
training modules (Safarik, 1987). Figure 3 provides an 
illustration of the types of information produced from 
this analysis and used in planning for the ftold-test 
program. 

************************* V* 

Insert i*igure 3 about here 
************ <•************** 
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The information shown in Figures 2 and 3 portrays the 
types of data obtained from the expert panel review and 
the analysis procedures used. While detailed presentation 
of both analyses go beyond the scope of this paper , readers 
interested in this information (i.e., Albright & Safarik, 
1986; Safarik, 1987) may obtain it from the senior author 
of this paper. 

Use of Results 

The summary analyses from the expert panel review 
were prepared by project staff members at California State 
University, Long Beach and subsequently shared with other 
key project personnel at the January, 1987 University of 
Georgia meeting. This information provided the basis for 
structuring the content and format of the training curriculum 
and for specifying the types of participants needed at 
the field-test sites to complete the training curriculum. 
Listed below are some of the central decisions made at 
the University of Georgia session: 

1. Sequence of Training Curriculum : The revised 
module sequence, as per recommendations of tl.e 
expert panel review, was adopted. That is, instead 
of a series of nine modules originally proposed, 
the training curriculum would consist of seven 
modules. It was also decided that an introductory 
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guide be added to this series which would describe 
the rationale for and characteristics of curriculum- 
based vocational assessment, 

2. Format of Training Curriculum ; The format for 
each module in the series would be performance-based 
in design and consist of the following six elements: 
(A) statement of module goal and competencies 
to be attained; (B) introductory section presenting 
rationale and purpose; (C) procedural section 
containing primary content of module; (D) references 
pertaining to module content; (E) example documents 
of assessment activities described in procedural 
section; and (F) suggested assignments to implement 
the content contained in the module. 

3. F ield-Test Participants ; As per the recommendations 
of the expert review panel , the following personnel 
types should be participant:!* at each in-service 
field-test program: (A) vocational special needs 
coordinators; (B) vocational instructors; (C) 
secondary special educators; (D) guidance 
counselors ; and ( F ) administrators of vocational 
education programs for /students with handicaps. 

Other important decisions, such as data collection procedures 
for the in-service field-test program, were also finalized 
at the University of Georgia meeting. These are described 
in the next section. 
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Phase II: Process Evaluation 



Introduction 

Process evaluation is basically a set of procedures 
used to monitor the i? iplementation of a planned program 
or product. Its main use is to obtain feedback that can 
aid staff in making decisions relative to program/product 
improvement ( Stuff lebeam, 1983). 

In the present project, process evaluation activities 
occur during the in-service and preservice testing of the 
training curriculum. These activities involve the collection 
of feedback from trainees, trainers, external product 
reviewers and project staff. 

Method 

As shown in the process evaluation design information 
on Table 2, multiple data collection strategies are used 
to obtain evaluative feedback on the training curriculum. 

*************************** 

Insert Table 2 about here 
*************************** 

The description of data collection procedures that follows 
is organized according to the sources for obtaining these 
data. 
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1. Field-Test Participants 

Trainees at the three in-service sites and the 
six university preservice settings evaluate the quality 
of individual training module . After reading module 
content and completing the respective in-service or 
preservice assignment , each trainee completes the 
CBVA Module Evaluation instrument • Thi s instrument , 
which was adapted from earlier product development 
studies by Albright (1980) and Phelps (1976), seeks 
trainee feedback on such variables as appropriateness 
of objectives, clarity of instructions, adequacy of 
suggested strategies, sequencing of module content, 
and accuracy and comprehensiveness of information 
presented in each module. A copy of this instrument 
appears in Appendix A. 

The CBVA Module Evaluation instrument is the 
primary mechanism being used for collecting trainee 
feedback on each of the training modules* However, 
a second instrument, titled the Product Asses sment 
Checklist , also adapted from Albright (1980) and Phelps 
(1976) is used to assess the quality of trainee products 
developed as a result of completing each module. 
This checklist actually has three uses: (1) as the 
trainee's guide during product development; (2) as 
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an evaluation tool for the trainer/course instructor; 
and ( 3 ) as another information source for the product 
developer to use in examining participant understanding 
of module instructions and guidelines . A sample 
checklist for Module 3 of the training curriculum 
is displayed in Appendix B. 
Trainsrs 

In-service and preservice trainers provide feedback 
on the training modules and the training context. 
An instrument titled the Trainer ' s Log is completed 
by the trainer following training on each module. 
This instrument, which is shown . in Appendix seeks 
information about the training arrangement and 
reflections on such variables as trainees response, 
quality of products developed, and needed improvements 
in training content. 

Evaluative data on the training curriculum will 
also be obtained from the trainers at a program review 
session to be held at George Washington University 
on May 23 and 24, 1988. More specifically, the trainers 
will provide written recommendations for improving 
the training products and for using the curriculum 
in preservice and in-service training programs. In 
addition, the trainers will provide suggestions for 
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the contents of a trainer's manual which will accompany 
the eight training modules. 
External Product Reviewers 

A third source for evaluative feedback is a group 
of six external reviewers. These individuals are 
university professors in vocational special needs 
education who are not involved with the field-test 
program at the in-service and preservice sites • Using 
a modified version of the CBVA Module Evaluation 
instrument, the reviewers are critiquing each module 
in the series according to such criteria as clarity 
and completeness of information, appropriateness of 
objectives , and suitability for intended training 
audiences. The external reviews of all training modules 
are scheduled to be completed by May 1, 1988. 
Project Staff 

Three activities were built into this process 
evaluation phase. First, an on-site observation of 
each in-service program was completed. These 
observations provided an opportunity for direct 
interaction between product developers and product 
consumers and for obtaining a richer contextual 
understanding of the use of these products in different 
settings . Second , continual communications between 
project staff and field-test trainers have been main- 
tained via telephone conversations, mail correspondence 
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and mutual attendance at professional meetings. While 
most of these discussions have been concerned with 
organizing, operating and evaluating the training 
program, informal assessments of program progress 
are usually woven into the conversations. Third, 
the scheduled two-day program review session in May, 
1988, with all of the preservice and in-service 
trainers, will provide a means for obtaining additional 
data for revising the training curriculum. 
Data Analyses and Interpretation 

Evaluation data received from field-test trainers 
and trainees and external product reviewers will be analyzed 
at two levels. The first level, micro-analysis, v/ill be 
used to analyze data for each training module. Data 
collected from trainees, trainers, and external product 
reviewers will be compared for each module to identify 
discrepancies or corroborative evidence (Fehrenbacher , 
Owens & Haenn, 1976; Phelps, 1976). Such variables as 
type of training setting (in-service or preservice) and 
respondent types (e.g., vocational instructor, administrator) 
will be examined to determine patterns and feedback for 
different applications of module use. 

In order to perform this micro-analysis, a large volume 
will need to be reviewed and synthesized. For example, 
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an estimated 120 in-service and preservice participants 
will have Cv..npleted 8 module evaluations each, providing 
nearly 1,000 instruments for review. In addition, an 
estimated 75 trainer's logs and an equal number of product 
assessment checklists will need to be analyzed and inter- 
preted. To complete these analyses, the narrative comments 
on these instruments will be transcribed verbatim into 
word processing files and resorted into four units of 
analysis: (A) level of use (e.g., in-service or preservice); 
( B ) type of respondent (e.g., administrator , guidance 
counselor); (C) module title; and (D) location of respon- 
dents (e.g., Minnesota, Georgia). Interactions across 
these units of analysis will also be analyzed. 

Once these narrative comments have been loaded and 
sorted into files, a qualitative data ^ analysis system 
(Seidel, Kjolseth & Clark, 1987) will be used to code the 
comments, aggregate comments, and generate summaries. 
These summaries will then form the basis for recommended 
changes at the individual module level, and for development 
of the trainer's manual. 

The second level of analysis, macro-analysis, will 
be used to compare and contrast reviewers' feedback relative 
to how well each of the individual modules appears to inte- 
grate into a comprehensive systematic process. This 
macro-approach, also used by Albright ( 1980 ) and Phelps 
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(1976), should be helpful in insuring consistency in the 
training format, and for making the training curriculum 
more fluid. 

Decisions relative to revision needed in the training 
curriculum will be made during June, 1988. A data matrix 
displaying the recommended changes from the field-test 
participants and the decisions made by the product writers 
will be prepared and sent to two independent reviewers. 
These reviewers will be asked to compare field-test 
recommendations to the decisions reached by the product 
authors. This independent review procedure will help ensure 
that the proposed changes in the training curriculum are 
consistent with the recommendations received via the 
field-test program. The individuals selected as independent 
reviewers must be familiar with th^ purpose and design 
of the project. Therefore, they will be selected from 
the expert review panel used in Phase I and/or the external 
product review group used in Phase II. 

Use of Results 

Data analyses and decisions relative to curriculum 
revisions will be completed in July, 1988. Revisions to 
the curriculum will be made by September. It is anticipated 
that the revised curriculum will be sent to the publisher 
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in late September and ready for distribution in November 
or early December, 1988. 

Phase III: Impact Evaluation 

Introduction 

Impact evaluation occurs following the training program. 
This evaluation phase examines the outcomes and effects 
of a program, focusing on two central and related questions 
(Brethower & Rummler, 1977): 

1. To what extent are the skills taught in the train- 
ing program being used by the graduate in the workplace? 

2. How is the application of these skills affecting 
the work environment? 

Impact evaluation of the present project will address 
these questions. In addition, evaluative data on the 
dissemination activities scheduled for the third and final 
project year will be collected. 

Method 

The <^esign for conducting evaluation activities during 
the 1988-89 academic year is shown in Table 3. Two 
activities will take place: 

*************************** 

Insert Table 3 about here 
*************************** 

(1) a follow-up study of personnel who participated in 
the in-service and preservice programs; and (2) an evaluation 
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of the third year dissemination program. Each is described 

below? 

1. Follow-- up Study 

A series of on--site observations and interviews 
will be conducted with former participants of the 
in-service and preservice programs. As suggested 
by the questions posed in Table 3, the purpose of 
this study is threefold : ( 1 ) to determine the extent 
to which former trainees have implemented curriculum^ 
based vocational assessment procedures in their 
programs; (2) to examine the impact of these procedures 
on system policy and practices; and (3) to assess 
trainee satisfaction with the preparation received 
through the field-testing program. 

Different data collection procedures will be 
used to study impacts at the in-service and the 
preservice levels. On-site observations and interviews 
will be conducted at the three in-service sites. 
The return to these sites will e:aable the project 
staff to obtain a richer contextual picture of impacts, 
via structured and unstructured observations and 
discussions with a variety of personnel at these 
locations (e.g., former trainees, students, admin- 
istrators). However, since the preservice program 
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is conducted at six universities and an estimated 
90 trainees from many schools and agencies are involved, 
it is anticipated that telephone interviewing with 
a stratified sample of former trainees from each 
university program will be the basic data gathering 
strategy. The follow--up study will be conducted by 
the project staff at California State University, 
Long Beach. 

2» Evaluation of Dissemination Workshops 

Since this evaluation will occur during the same 
timeframe as the follow-up study, it is identified 
in Table 3. However, in terms of its purpose, it 
is a process evaluation activity. Data collected 
about the types of participants attending these 
dissemination workshops and feedback on the quality 
of the workshop program from participants and trainers 
should be especially helpful in making decisions about 
subsequent product dissemination and marketing 
strategies . 
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Discussion 



At this point in the project, the field-test program 
is nearing completion and the data are being collected 
and organized for analysis. As the data analysis stage 
nears, it seems that the concern about employing precautions 
to reduce developer bias heightens. Whil6 such concern 
is warranted ^. ^ the data analysis and interpretation stages 
of product revision, precautions to limit developer biases 
should prevail throughout the operation of a project. 
The use of the three-phase evaluation framework in examining 
the various points at which independent and multiple reviews 
need to built into the curriculum development, testing 
and revision process has been helpful in this regard. 

In the attempt to obtain indepenaent feedback on the 
quality and use of the training curriculum, the field-test 
program is being conducted at multiple training sites with 
most being directed by teacher educators who are not involved 
in the development of the training products. From an 
evaluation design perspective, this particular approach 
is attractive and desirable for reducing developer influence. 
However, it also introduces a^ixtional demands or 
considerations that the project administrator should be 
prepared to address. Three in particular are mentioned 
below. 
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First, by taking the program out to multiple locations 
with many personnel involved in product testing, the news 
about these products rapidly spreads. If the news is 
positive , numerous ^ equests for copies o'£ the products 
and/or additional project information are likely to mount 
as the project progresses. The administrator may be inclined 
to share these training products. However, the primary 
purpose of the field test program must be mainto.t ned.. 
That is, the testing of the products is to examine their 
utility in applied settings and subsequently refine these 
products to enhance utility* Product dissemination comes 
after, not during the field-test program. 

Second, the selection of independent trainers requires 
adequate time for preparing these persons to mount and 
conduct the field-test training program. A thorough 
orientation to project goals, training curriculum content, 
and field-test requirements is an important beginning point. 
However, additional time with these trainers will be 
necessary as the field-test program gets underway anu the 
variety of questions about training arrangements , training 
content and data collection procedures come forth* 

The third and final consideration relates to the 
validity of the field test process. Since the primary 
purpose of this process is to obtain independent evaluative 
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feedback from the various audiences associated with the 
training curriculum, external validity must be maximized . 
Unlike research in a classic sense, an evaluative study 
of this nature will need to sacrifice elegance of design 
so that recommendations for program improvement are broadly 
generalized. However, some controls for internal validity 
need to be in place. A recent site visit by one project 
member illustrates this point. The purpose of this visit 
was to observe the training program in progress. Yet, 
when the staff member arrived at the site, both trainer 
and trainees fully expected him to take an active lead 
in the training activities. After clarifying his purpose 
with both parties, the independent observer posture was 
maintained. 
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Figure 1 

Sample Format of a Micro-Review Instrument 



Module 4: Placing Students in Vocational Education Programs 

Purpose: This module will provide the trainee with 

instruction on assessments to be employed with 
a student being considerad for placement in a 
specific vocational program. 

Section I: Competencies Rating 



This section consists of a series of competency 
statements. You are asked to rate the impor- 
tance of each competency for the four personnel 
groups listed. Please circle the rating number 
for each personnel group listed. 

Rating Code ; 

1 Yes, definitely include as part of the train- 
ing module for this group. 

2 No, not imperative or important for this 
group. 

3 Don't know, uncertain or have no opinion 
about the importance of this competency for 
this group. 



Example Rating Procedure : 



Don't 
Yes No Know 



A. The trainee will prepare a checklist 1 2 3 

for observing student performance in 
a vocational setting. 

Vocational Special Education Resource 

Teachers 12 3 

Secondary Special Educators 12 3 

Vocational Instructors 1 2. 3 

Vocational Counselors 12 3 



Section II: Review Comments: 

1. Please comment on the appropriateness and comprehensive- 
ness of the stated competencies for this module. 
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Table 1 

Phase I Planning Evaluation Design 



Purpose (s) Key Questions 



Timing 



• To obtain 1, 
feedback 
on compe- 
tencies or 
skills 
needed by 
different 
personnel 
to success-2, 
fully 
implement 
the vari- 
ous compo- 
nents of a 
CBVA 
process 



How important are 
the identified 
competencies in 
order to implement 
each component of 
the CBVA process? 



How important are 
each of the compe- 
tencies to each of 
the various indivi- 
duals who impact 
upon the implemen- 
tation of the 
CBVA process ? 



During the 
beginning 
stages of 
project, from 
month 4 to 
month 6 of 
the project . 



Instrumentation Data Sources 



Survey instru- 
ment focusing 
on each module 
and associated 
personnel 
competencies 
and roles. 



Expert panel 
review of 
selected 
modules . 
Each panel 
member 
reviews two 
modules . 



To deter- 1. 


How comprehensive 


During the 


A survey 


mine 


does the CBVA 


beginning 


instrument 


appro- 


system appear 


stages of 


examining 


priateness 


to be? 


project, from 


overall 


and cohe- 




month 4 to 


training 


siveness 2, 


How logical does 


month 6. 


system. 


of 


the sequencing of 






elements/ 


of module compo- 






components 


nents and 






of the 


accompanying 






planned 


competencies 






CBVA 


appear to be? 






process. 









Expert panel 
review of 
complete set 
of module 
components 
and personnel 
competencies . 
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Figure 2 

Data Analysis and Interpretation Format 



Level of Analysis Recommendation 



Decision Made 



Rationale for Decision 



I. MACRO-ANALYSIS 
(System Review 
Comments ) 



A. Sequence 
of Modules 



Module 7 

( Individual Program 
Review)' 

is a bit thin. 
Could be split 
between Module 6 
(Monitoring Student 
Progress) and Module 
8 (Transitional 
Service Assessment). 



Delete Module 7 



from system, 
incorporate 
appropriate 
competencies 
into Modules 
and 8 . 



but 



By 



Module 7 consisted of 
competencies that over- 
lapped with purposes 
of Modules 6 and 8. 
integrating Module 7 
competencies into Modules 
6 and 8, a more logical 
and functional presenta- 
tion is possible. 



Module 
Titles 



Change "program" 
to "process" in 
Module 1 title. 



Revise title to Change more accurately 

read "Establish- reflects process 

ing a Curriculum- emphasis of .system. 

Based Vocational 

Assessment 

Process." 



C. Complete- 


Module related to 


Content of 


Priority and importance 


ness of 


transitional services 


.module will 


of transiif.ional services 


Competency 


needs to be tied to a 


address this 


delivery deserves 


Statements 


framework specifi- 
cally for transi- 
tional services . 


concern. 


special attention in 
in this module. 



Note: Extracted from L. Albright, and L. Safarik, (December, 1986), Summary 



analysis, expert panel review results, curriculum-based vocational assessment system. 
Long Beach, CA: California State University, Long Beach, Bureau of Employment- 
Related Education and Training for Special Populations. 
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Figure 3 

Personnel Roles in CBVA System: Summary of Results from Expert Panel Review 



Module Title 



Personnel 



Considerations 



Module 1: Establishing a 
Curriculum-Based Vocational 
Assessment Process 



Module 4: Planning the 
Student's Voccitional 
Program 



* Lead Person 



*1, Vocational Special 
Education Resource 
Teacher (VSERT) 

2. Secondary Special 
Educator (SSE) 

3. Vocational Instruc- 
tor (VI 

4 . Vocational Counselor 
(VC) 

5 . Administrative 
Personnel 

6 . Paraprof essionals 



Panel review data indi- 
cates a need for an 
emphasis on teaming in 
Module 1. Involvement of 
all personnel essential 
at this initial phase. 

The involvement of key 
administrative personnel 
(both SE and VE) is 
necessary in Module 1. 

The need for the 
involvement of para- 
professionals in later 
modules is indicated and 
therefore, they should be 
included in the establish- 
ment of the process in 
Module 1. 



1. VSERT 

2. SSE 

3. VI 

4. VC 



The VSERT and SSE were 
both viewed as key 
figures in this module. 

The involvement of para- 
professionals would be 
important for competen- 
cies D & E 



Note: Extracted from L. Safarik, (January, 1987), Personnel requirement summary 
by module . Long Beach, CA; California State University, Long Beach, Bureau of 
Employment-Related Education and Training for Special Populations. 
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Table 2 

Phase II Process Evaluation Design 



Purpose 



Key Questions 



Timing 



Instrumentation 



Data Cources 



To obtain 

feedback 

to improve 

training 

products 

and 

processes 



1. To what extent 
are each of the 
modules appro- 
priate in terms 
of length, 
clarity, 
accuracy and 
internal 
consistency? 



2. To what extent 
do the products 
developed by 
participants 
for each module 
reflect stated 
performance 
criteria? 



During in- 
service and 
preservice 
field-test 
programs 
(i.e. , April, 
1987-May, 
1988) 



• CBVA Module 
Evaluation 
instrument 



« Trainer's 
Log 

o Trainer ' s 
Program 

Review Reports 

• Performance 
Assessment 
Checklist 

• Trai ner ' s 
Log 



• Trainees of 
the in- 
service & pre- 
service pro- 
grams (N=120) 

• External pro- 
^duct reviewers 

(N = 6) 

• Field-site 
trainers (N_=6) 

• Trainee 
products 



^ Field-site 
trainers 
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3. In what train- 
ing contexts do 
the modules 
appear to be 
most effective, 
both from an 
individual 
module perspec- 
tive and in 
terms of over- 
all training 
curriculum? 



o Trainer's 
Log 

a Notes from 
project staff 
on-site obser- 
vations and 
field-test 
program review 
session. 

<» Trainer's Program 
Review Reports 



• Field-site 
trainers 

• Project 
staff 
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Table 3 

Phase III: Impact Evaluation Design 



Purpose Key Questions Timing Instrumentation Data Sources 



To determine 
the changes 
in systems & 
individual 
practices 
that are 
associated 
with use of 
training 
program • 



To determine 
the most 
effective 
approaches to 
disseminate 
information 
about the 
CBVA system. 
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1. To what extent 
have local sys- 
tems changed 
their assessment 
policies & prac- 
tices as a result 
of their involve- 
ment & training 
with CBVA? 

2. To what extent 
has the train- 
ing in CBVA 
changed the 
assessment 
practices of 
individuals 
who have been 
have been 
trained in its 
use? 

3. What is the 
best dissemina- 
tion/training 
milieu (e.g. , 
types of 
trainees , num- 
ber of trainers, 
format of train- 
ing to most 
effectively 
diffuse the CBVA 
system to poten- 
tial adopters? 



Starts about 
the beginning 
of the third 
project year 
& continuous 
until the end 
of the project 



Starts about 
the fifth 
month of the 
3rd project 
year and 
continues 
until the end 
of the project* 



« Trainee entry- 
level- 
surveys 



• On-site inter- 
views with 
participants 
and trainers 
of three in- 
service 
programs . 

• Telephone 
interviews 
with pre- 
service 
trainers and 
trainees 



• Evaluations of 



Population of 
individuals 
involved in 
the three in- 
service train- 
ing activities 



Sample of 
partici- 
pants from 
each pre- 
service 
site 



Summary 



each of the pre- comments of 
sessions at AVA presession & 
and CEC con- workshop 
ventions . trainers • 



o Evaluations 
of the four 
regional 
training 
workshops 



Population or* 
of participants 
involved in the 
presessions and 
training work 
shops • 
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Appendixes 




Appendix A 
CBVA Module Evaluation Instrument 
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CBVA MODULE EVALUATION 



Date^ 

Module # & Title 

Reviewer Name Position Title 



Note to Reviewer: This instrument is the primary source used to 
obtain participant feedback on the quality of each module in the 
CBVA training program. /Your candid and complete response to the 
information requested in the four parts of this instrument will 
be helpful in determining the changes needed to improve the 
quality of each module. We thank you in advance for taking the 
time necessary to contribute to this effort. 



Part I: Time Investment 

Please indicate the amount of time spent in completing 
the following module activities: 

Activity Time Spent 



A. Reading module content 



B. Discussing information and 
procedures in module 



C. Organizing for team completion 
of in-service assignment 



D. Completing in-service assignment 



E. Completing written evaluation of 
module 



Total Time = 



Part II: Evaluation of Module Components 



Listed below are the major components of the module you 
have reviewed. We would like your evaluation of each 
component. Several items to consider when reacting to 
each element include: 



e Length - 
® Clarity - 
^ Accuracy - 



Is the component too lengthy or does it 
it need expansion? 

Is the information presented easy to 
understand or is it confusing? 

Is the information on target with the 
competencies of the module? Is the 
information accurate and current? 



® Sequence - Is the component situated in a logical 

order with the rest of the components or 
should it be placed elsewhere? 



Module Components 
Goal and Competency Statements: 



Introduction J 



3. Strategies; 



4. Example Documents: 



5. In-Service Assignment: 
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Part III: Module Reorganization: 

The information you provide to the following items 
will be helpful in reviewing the adequacy of the 
present module and in determining what changes need 
to occur. 

Please check the appropriate category for your 
response to each item and^ when necessary^ provide 
specific recommendations for improving the module. 

1. How appropriate are the competencies for this module: 

Very Appropriate Revisions Needed Undecided 

Recommendations : 



2. The strategies presented in this module are related directly 
to the module competencies: 

Agree Disagree Undecided 



Recommendations ; 



3. The strategies presented in the module are appropriate 
. and realistic. 

Agree Revisions Needed Undecided 

Recommendations : 
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4. The information presented in this module is easy to follow 
and in a logical sequence. 

Agree Revisions Needed Undecided 



Recommendations : 



5. The length of this module is appropriate. 

Agree Revisions Needed Undecided 

Recommendations : 



Part IV: In-Service Arrangement 

This section is concerned with your overall reaction 
to the in-service arrangement used in completing this 
module. Specific comments and recommendations for 
improvement would be most helpful. 

1. Please comment on the adequacy of the in-service arrangement 

for discussing module content and in completing module activities. 



2. Please comment on your experience in completing this module. 
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Appendix B 
Product Assessment Checklist 
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PRODUCT ASSESSMENT CHECKLIST 



CBVA #3: Placing Students in Vocational Education Programs 

Listed below are the criteria to be used by the 
instructor in evaluating each of the completed 
activities within Module 3 • When an in-service 
activity is completed, the product of the activity 
(e.g. , listing of available service, observation 
rating scale) should be submitted to the in-service 
instructor along with your copy of this checklist • 
The product will be evaluated and returned to you 
as soon as possible. 

Each completed activity will be rated according 
to a three-part scale, A rating of 1 indicates 
excellence; a 2 rating is considered satisfactory; 
and a 3 rating means the product is in need of 
further improvements. 



Name_ School 

In-Service Assignment #1: Exploratory Assessment Team Process 

<a A collaborative teaming process was used in 

completing this activity, 12 3 



o i'he plan for one student's exploratory program 
is fully documented according to the guidelines 
shown in Example Document #2, 12 3 

In-Service At^signment #2: Records Review 

m The student records reviev; was done in relation 

to the areas outlined in Example Document #1. 12 3 

• The review summary is comp3-etG and thorough, 12 3 

In-Service Assignment #3: Basic Skills Testing 

© The test or tests selected was verified by 

vocational education personnel within the district 12 3 
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« A reporting of the skills testing try-out with 
students indicates that a thorough analysis of 
the testing process was completed. , 12 3 

In-Service Assignment #4: Vocational Program Inventory 

<9 The completed inventory suggests that a thorough 

review was conducted. 12 3 

o The try-out in two programs produced information 
about the usefulness of this' inventory for planning 
a student's program, . 12 3 

^30 



Product Assessment Checklist (continued) 

In-Service Assignment #5: Idr ratification Test 

e The identification test was developed in 
collaboration with appropriate vocational 

education personnel. 12 3 

0 The results of a try-out with two students were 
used to improve the usefulness of this testing 
procedure. 12 3 

1 .-Service Assignment #6: Performance Sample 

o The performance sample was developed collaboratively 

by vocational and special education personnel. 12 3 

0 The student instructions for completing the 

performance sample are clearly written and easy 

to follow. 12 3 

o The instructions for persons administering the 

sample are also clear and easy to follow. 12 3 

© The performance sample is a realistic measure 

of important program skills. ' 12 3 



Date Submitted: Returned: 

Total Score: 



Comments : 
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Appendix C 
Trainer's Log 
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TRAINER LOG 



Date 

Module # & Title Trainer Name{s) 



This log is to be done after you and the training group 
complete a particular module. 

Part I: Description of In-Service Arrangement 

1. Please provide a short description of how the training group 
completed this module. That is, describe who, by position 
title, participated in module activities and how they 
organized for completion of the in-service assignment. 



2, Please describe the frequency and type of contact with 
trainees during module completion (e,.g,, number of group 
sessions, technical assistance provided to individua-ls ) • 
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Part II: Reflections 



After working with the group on this module, please 
reflect on the following items: 



I. Training Group Response to Module Content and Activities: 



II. Quality of Products Developed by Trainees: 



III. Personal Reactions/Feelings to this Training Experience: 



IV. Recommendations for Improving Training Content and/or 
Process: 
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